In Herschel images of the Galactic plane and many star forming regions, a major factor limiting our ability to extract faint compact sources is cirrus confusion noise, operationally defined as the "statistical error to be expected in photometric measurements due to confusion in a background of fluctuating surface brightness". The histogram of the flux densities of extracted sources shows a distinctive faint-end cutoff below which the catalog suffers from incompleteness and the flux densities become unreliable. This empirical cutoff should be closely related to the estimated cirrus noise and we show that this is the case. We compute the cirrus noise directly, both on Herschel images from which the bright sources have been removed and on simulated images of cirrus with statistically similar fluctuations. We connect these direct estimates with those from power spectrum analysis, which has been used extensively to predict the cirrus noise and provides insight into how it depends on various statistical properties and photometric operational parameters. We report multi-wavelength power spectra of diffuse Galactic dust emission from Hi-GAL observations at 70 to 500 microns within Galactic plane fields at l= 30 degrees and l= 59 degrees. We find that the exponent of the power spectrum is about -3. At 250 microns, the amplitude of the power spectrum increases roughly as the square of the median brightness of the map and so the expected cirrus noise scales linearly with the median brightness. Generally, the confusion noise will be a worse problem at longer wavelengths, because of the combination of lower angular resolution and the rising power spectrum of cirrus toward lower spatial frequencies, but the photometric signal to noise will also depend on the relative spectral energy distribution of the source compared to the cirrus.